Reference: Biol. Bull. 194: 100-107. (April. 1998) 


Concepts in Imaging and Microscopy: 
Color Image Processing for Microscopy 


KENNETH R. CASTLEMAN 

Vice-President for Research , Perceptive Scientific Instruments, Inc , 2525 South Shore Blvd. } 

League City , Texas 77573 


Abstract. Digital image analysis has long been applied 
to monochrome images from the light microscope. Using 
these techniques one can locate, measure, identify, and 
count the objects of interest in a microscope field of view. 
Since microscope images often contain color information 
of interest, it is now becoming more common to analyze 
the spectral content of images as well. Although most of 
the digital imaging techniques developed for mono¬ 
chrome microscopy work well with color images, there 
are a few considerations specific to the analysis of multi- 
spectral imagery. These include obtaining a properly bal¬ 
anced multicolor digital image, specifying the color com¬ 
ponents of the image, and compensating for imperfect 
separation among the color channels. 

Introduction 

In two-dimensional digital image processing for mi¬ 
croscopy, images are commonly considered to have a 
single scalar gray level (brightness or optical density) 
that is a function of two spatial coordinates. With color 
imaging there are two or more gray levels per pixel, each 
corresponding to a different spectral band. When we pro¬ 
cess three spectral bands corresponding to the red, green, 
and blue colors to which the human visual system re¬ 
sponds, we call it “color image processing/' A tri-color 
image can be formed by separately sampling three spec¬ 
tral bands over the two spatial coordinates of an optical 
image. In this article we focus on tri-color image analysis, 
although the generalization to four or more colors is 
straightforward. 

The legacy of human vision 

The retina of the human eye is covered with photore¬ 
ceptor cells that are functionally analogous to the sensor 


sites (picture elements, or “pixels") on the CCD chip in 
a television camera. These cells absorb light from the 
image formed on the retina by the lens and cornea, and 
they send nerve impulses to the brain. The photoreceptor 
cells are a mixture of two types, rods and cones. While 
the more sensitive rods provide night vision, the cones 
afford color vision at higher light levels. 

The cones occur in three types, each having a different 
spectral sensitivity. They divide the visible portion of the 
electromagnetic spectrum into the three bands that we 
perceive as red, green, and blue (RGB). These, then, are 
the three primary colors of human vision. Figure 1 shows 
the sensitivity spectra of the three types of cones in the 
human visual system (Wald, 1964). Notice the significant 
degree of overlap. 

Because human vision uses three color channels, the 
bulk of the effort in the development of electronic im¬ 
aging hardware has been devoted to tri-color systems. 
This is particularly true for television cameras, image 
digitizers, display systems, and image printers. As a re¬ 
sult, tri-color equipment is produced in high volume and 
is widely available at relatively low cost. Digital image 
analysis techniques can be applied to images having any 
number of primary color channels, but the tri-color model 
is often used, simply because of hardware availability. 
We focus here on tri-color imaging to illustrate the most 
important points. In practice, however, the nature of the 
problem should dictate how' many color channels are 
used. 

Color digital image analysis 

A monochrome digital image can be thought of as a 
scalar function of two spatial dimensions (a\ y). By exten¬ 
sion, a color digital image can be viewed as a scalar 
function of three dimensions, two spatial and one spectral 
(.v, v, \). It is usually more convenient, however, to treat 
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Figure t. The speciral sensilivity of the cone photoreceptors in the 
human eye (after Wald, 1964). 


it as a two-dimensional image having multiple gray levels 
(e.g. t red, green, and blue) at each pixel instead of only 
one. In other contexts it is more useful to consider it 
to be an overlay of monochrome digital images, each 
representing one of the primary colors. (We refer to these 
overlays as color component images or as the color chan¬ 
nels .) The latter two viewpoints simplify the processing 
and analysis of color images because many of the tech¬ 
niques commonly applied to monochrome images can 
then be applied to color images with little modification. 

Microscope image analysis 

Three types of microscopic specimens exhibit color. 
Some specimens have color intrinsically. Examples are 
red blood cells and organic mineral samples. Other speci¬ 
mens take on color only as a result of staining or other 
chemical preparation prior to examination. A third cate¬ 
gory includes specimens given color by the optics of the 
microscope (polarization, birefringence, etc.). 

In biomedical work, different chemical constituents of 
a specimen are often stained with absorbing or fluorescent 
dyes of different colors. In many applications the different 
structures are separable by color, and one wishes to visu¬ 
alize or measure these objects separately and in correct 
spatial relationship to one another. 

If, for example, three chemical components of a speci¬ 
men are stained, separately, with red, green, and blue 
fluorescent dyes, one can digitize and display a normal tri¬ 
color image of the specimen. The three color component 
images are registered monochrome images, each of which 
shows objects of a specific type. This paves the way for 
isolating, measuring, and classifying the objects by using 
standard techniques for the analysis of digital images. 

Commonly, the absorption or emission spectra of the 
stains or fluors do not match the curves in Figure 1. Then 
the red, green, and blue primary colors of human vision 
(and of commonly available hardware) are not ideal for 
isolating the differently colored objects in the specimen. 
When it is necessary to display the specimen in its natural 


appearance, one must account for the spectral response 
of the camera and of the display unit in order to render 
an accurate-appearing reproduction. On the other hand, 
quantitative multispectral image analysis can be done us¬ 
ing any set of (two or more) primary colors. In this case 
it is often advisable to abandon the RGB system and 
select primary colors that match the stains or fluors in 
use. 

Color Balance 

Often a digitized tri-color image will not render the 
various colors properly when displayed. Nonuniform illu¬ 
mination, combined with differing sensitivities and offsets 
(black levels) in the three color channels, can unbalance 
the three color-component images so that the objects in 
the field appear to be shifted in color. Most noticeably, 
objects that should be gray take on unwarranted color. 
The first test of proper color balance is that all the gray 
objects indeed appear gray. The second test is that the 
highly saturated (bright) colors must all have the proper 
hue (defined below). 

The remedy for color imbalance is to use linear 
grayscale transformations (Castleman, 1996) on the indi¬ 
vidual red, green, and blue images. Normally, two of the 
component images are transformed to match the third, as 
follows. One selects two relatively uniform areas of the 
image, one light gray and one dark gray, and computes 
the mean gray level of both areas in each of the three 
component images. One then defines linear grayscale 
transformations (i.e., y = ax + b) to make two of them 
(e.g.y red and blue) match the third (c.g., green). One must 
solve two linear equations in two unknowns to determine 
a and b. This forces each of the two areas to have the 
same gray level in all three component images, and this 
will usually achieve color balance. If the camera and 
digitizing system are stable, the required transformations 
can be determined in advance by digitizing a black-and- 
white test target. The resulting transformations can then 
be used routinely throughout the digitizing session. 

Color Specification 

There are several ways to specify, quantitatively, the 
color of a pixel in a tri-color digital image. Here we 
discuss two of the more useful methods. 

RGB format 

The most common color specification simply uses the 
red, green, and blue brightness values scaled, tor example, 
between zero and one. This convention is called 'RGB 
format.” The color of each pixel can be represented by 
the location of a point in the first quadrant of three-dimen- 
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sional color space (RGB-space). This is shown by the 
color cube in Figure 2. 

The point at the origin of RGB-space represents zero 
brightness of all the primary colors and is thus the color 
black. Full brightness of all three primaries together ap¬ 
pears as white at the comer diagonally opposite the origin 
of RGB-space. Three of the corners of the color cube 
correspond to the primary colors, red, green, and blue. 
The remaining three corners correspond to the secondary 
colors, yellow, cyan (blue-green), and magenta (purple). 
Equal amounts of all three color components at lesser 
brightness produce a shade of gray. The locus of all such 
points falls along the “gray line," which is the diagonal 
of the color cube connecting the black and white points 
(Fig. 2). 


HSI format 

Another useful specification scheme is called the “HSI 
format" (hue, saturation and intensity). It reflects the way 
humans perceive color, and it offers advantages for color 
image processing as well (Castleman, 1996; Pratt, 1991: 
Russ, 1995). 

In HSI format the “1" stands for intensity, or bright¬ 
ness. It is, for our purposes, simply the sum of the R, G, 
and B gray level values, although different schemes with 
unequal weighting of the colors are also used. The inten¬ 
sity value specifies the overall brightness of the pixel, 
without regard to its color. One can convert a color image 
to monochrome by computing only the intensity at each 
point, thereby discarding the color information. 

The color information is contained in two parameters 
that are called “hue" and “saturation," although other, 
equivalent terms are often used. Hue and saturation are 


Cyan 


White 



Yellow 


Figure 2. RGB color space. Each of the primary colors defines one 
axis of a three-dimensional coordinate system. Every color plots to a 
position in this space. 



Figure 3. The color circle. Hue is an angle, and saturation is the 
radial variable. 


illustrated by the color circle in Figure 3. The hue of a 
color refers to which spectral wavelength (color of the 
rainbow) it most closely matches, and is expressed as an 
angle. Arbitrarily, a hue of 0° is red, 120° is green, and 
240° is blue. Hue traverses the colors of the visible spec¬ 
trum as it goes from zero to 240 degrees. Between 240° 
and 360° fall the nonspectral (purple) colors that the eye 
perceives. 

The saturation parameter is the distance of the point 
from the origin of the color circle. The “pure" or “satu¬ 
rated" colors fall around the periphery of the circle, and 
their saturation values are unity. At the center of the color 
circle lie the neutral (gray) shades; that is, those with zero 
saturation. 

The concept of saturation can be illustrated as follows. 
Consider a bucket of bright red paint. The color has a 
hue of 0° and a saturation of unity. Mixing in some white 
paint makes the red less intense, thereby reducing its 
saturation, but without making it darker (/.c., reducing its 
intensity). Pink corresponds to a saturation of 0.5 or so. 
As more white is added to the mixture, the red becomes 
paler, and the saturation decreases, eventually ap¬ 
proaching zero (white). If, on the other hand, black paint 
is mixed with the bright red paint, the intensity of the 
color decreases (toward black) but its hue (red) and satu¬ 
ration (1.0) remain constant. 

The three color coordinates, taken together, define a 
cylindrical color space (Fig. 4). The gray shades fall along 
the axis from black at the bottom to white at the top. The 
fully bright, fully saturated colors fall on the perimeter 
of the top surface of the cylinder. 

Many other color coordinate systems are used. Those 
established by the Commission Internationale de V Eclair- 
age (CIE), an international standards committee for light 
and color, are perhaps the most widely used. They are 
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Figure 4. Cylindrical color space, tnlensity is added as the axiat 
variable. Every cotor plots to a position in this space. 


based on experimental data from color-matching experi¬ 
ments conducted on human observers. 

Color coordinate conversion 

Some image processing techniques are intrinsically 
more successful when carried out in rectangular RGB- 
space, whereas others work best in cylindrical HSI-space. 
Thus it is useful to be able to convert color digital images 
from RGB to HSI color coordinates and back again. 

RGB to HSI conversion 

The conversion from RGB to HSI format can be ap¬ 
proached as follows. Recall that the gray line is the diago¬ 
nal of the color cube in RGB-space, and it is the vertical 
axis in cylindrical HSI-space. Thus we can begin by rotat¬ 
ing the RGB cube in a, y, c-space so that its diagonal lies 
along the c-axis, and its R -axis lies in the a-z plane (Ledley 
et al. t 1990: Castleman, 1996). This is given by 

' =T6 [2R - G ~ B] 


>' = f2 lG - B ' 


fall on a unit-radius circle in the x-y plane. This leads to 
the formula for saturation 


V3 

S = 1 — — min (R. G. R) 

We compute the angle 

%l(R ~ G) + (R- B)} 


9 = cos 1 


_V(fl - G) 2 + (R - B)(G 


M 

- B )J 


( 2 ) 


(3) 


and the hue is 


H = 


0, G > B 
2n — 0, G < B 


(4) 


Notice that black (i.e., [R, G, B] = [0, 0, 0]) creates a 
problem for Eq. (3). We can assign black to [H, S, /] = 
10 , 0 , 0 ]. 

HSI to RGB conversion 

The formulas for converting from HSI to RGB coordi¬ 
nates take on slightly different form for different sectors 
of the color circle (Castleman. 1996). For 0° < H < 120°, 


*-J5 


1 + 


S cos (H) 
cos (60° — H) 


B = T 3 (] ~ S) 

G = Si - R - B 
while for 120° < // < 240°, 

S cos (H - 120°)' 


(5) 


I 


1 + 


cos (180° - H) 


R = - ( l-S) 

B = Hi - R - G 
and for 240° < H < 360° 


( 6 ) 




I + 


S cos (H - 


cos (300' 


- 240°) 1 
0 “ H) \ 


f3 


[R + G + B] = I 


( 1 ) 


The latter of these gives the HSI intensity parameter. 

We can convert to cylindrical coordinates by defining 
polar coordinates in the x-y plane. Then we normalize the 
radial variable so that the fully saturated colors (those 
having no more than two of the primary colors present) 


c =® (, - s> 

R = V37 - G - B 


(7) 


The RGB cube (Fig. 2) does not map directly into the 
color cylinder of Figure 4. Instead, it delines a somewhat 
cone-shaped region in HSI-space, with white at the apex. 
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Since the entire floor of the color cylinder is the single 
color black, some authors prefer to consider the HS1 color 
space to be biconic. 

There are several variations of the HSI conversion for¬ 
mulas (Pratt, 1991), but the particular one chosen usually 
will not materially affect the result, as long as (1) intensity 
is a weighted average of /?, G, and B\ (2) hue is an angle; 
(3) hue, saturation, and intensity are independent; and (4) 
the transformation is accurately invertible. 

Multispectral images having more than three primary 
colors can also be transformed into corresponding multi¬ 


dimensional color spaces. For example, one can convert 
from rectangular to spherical coordinates in a color space 
based on four or more primary colors. 

Color Compensation 

Available stains and fluorophores have different, often 
broad absorption and emission spectra, and commonly 
used color cameras have broad, overlapping sensitivity 
spectra. As a result, one seldom obtains complete isolation 
of three types of objects in the three component images. 



Figure 5. Three-color, fiuorescenlly labeled bone marrow cells. (A) Color image: (B) red; (C) green; 
(D) blue. Notice that all structures appear in each of the R. G. and B images. 
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Instead, each type of object is visible in all three color- 
component images, although at reduced contrast in two 
of them (Fig. 5). 

We can model the spreading of light among the color 
channels as a linear transformation (Castleman, 1993, 
1994, 1996). Let the matrix, C, specify how the colors are 
spread among the three channels in a three-color fluores¬ 
cence application (example below). Each element c tJ is the 
proportion of the brightness from fluorophore j that appears 
in color channel / of the digitized image. Let x be the 3 
by 1 vector, ( R , G , Z?j T , of actual lluorophore brightness 
values at a particular pixel, scaled as gray levels that would 
be produced by an ideal color image digitizer (one with 
neither color spread nor black level offset). Then 

y = Cx + b (8) 

is the vector of RGB gray levels recorded, at that pixel, 
by the image digitizer. While C accounts for the color 
spread, the vector b accounts for the black level offset of 
the digitizer. That is, b t is the gray level that corresponds 
to black (zero brightness) in channel /. Equation (8) is 
easily solved for the true brightness values by 

x = C - b] (9) 

Color spreading can thus be eliminated by first sub¬ 
tracting the black level from each channel, and then pre¬ 
multiplying the resulting RGB gray level vector for each 
pixel by the inverse of the color spread matrix. 

Color compensation example 

Figure 5 shows an RGE5 image of human bone marrow 
cells; the image was digitized with a single-chip CCD 
color television camera mounted on a fluorescence micro¬ 
scope. The camera’s sensitivity spectra were similar to 
those in Figure 1. The cells were counterstained with 
DAP1 (4',6-diamidino-2-phenylindole, dihydrochloride), 
which fluoresces in the blue. Dividing cells also picked 
up the green fluorophore, FITC (fluorescein-5-isothiocya- 
nate), which had been conjugated with BrdU (5-bromo- 
deoxyuridine). The centromeres of the number 8 chromo¬ 
somes were labeled with Texas red, a red fluorophore, 
attached to a DNA probe. 

Ideally, the cells would appear in the blue channel, divid¬ 
ing cells would show up in the green channel as well, 
and two dots per cell (corresponding to the two number 8 
chromosomes) would appear in the red channel. In Figure 
5, however, all components appear in all channels because 
of overlap among the fluorophore emission spectra and the 
sensitivity spectra of the camera’s three color channels. 

Table I shows the color spread matrix for the system 
used to digitize the image in Figure 5. The values in this 
matrix were determined experimentally from digitized 
images of cells stained with single fluorophores. They 
represent the relative contrast of different colored objects 


Table I 


Color spread matrix for the image in Figure 5 


Color channel 


Fluorophore 


Texas Red 

FITC 

DAPI 

Red 

0.85 

0.26 

0.24 

Green 

0.05 

0.65 

0.32 

Blue 

0.10 

0.09 

0.44 


The columns represent the three fluorophores used, and the rows 
represent the three channels of the color camera. The values in the table 
specify the proportion of the emitted light from each fluor that appears 
in each color channet. 

in each of the three color channels. Table 1 states, for 
example, that only 44% of a DAP1 molecule's brightness 
is recorded in the blue channel, while 32% of it shows 
up in the green channel, and 24% finds its way into the 
red channel. The color smear in this example is relatively 
severe, and use of a monochrome camera with separate, 
optimized, color filters, for example, can produce better 
results. But color smear is intrinsic to the imaging process 
and is always present to some degree. 

The inverse, of the matrix in Table I is the color 
compensation matrix. 



1.24 

- .45 

-.35 


c 1 = 

0.05 

1.69 

- 1.26 

( 10 ) 


-.29 

-.24 

2.61 



It specifies how, using Eq. (9), one can correct the color 
spreading. The values in the left column specify that one 
should, at each pixel, take 124% of the gray level in the 
red channel image, add 5% of the green channel value, 
and subtract 29% of the blue value to produce a corrected 
red-channel gray level. The second and third columns 
specify how to correct the pixels of the green and blue 
channels, respectively. 

Figure 6 shows the result of color compensation applied 
to the color image in Figure 5. Here the three differently 
stained types of objects are effectively isolated to the 
images in the red, green, and blue channels. This permits 
more accurate segmentation and measurement of the cell 
and its components. Because color spread tends to “wash 
out” an image, color compensation tends to increase both 
the contrast and the saturation of the displayed color im¬ 
age. Since it is intended to isolate the different colored 
components into separate color channels, it may not pro¬ 
duce visually realistic images upon display. 

Color Image Analysis 

Image segmentation 

Segmentation of a tri-color image can be viewed as a 
process of partitioning the three-dimensional (RGB or 
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Figure 6. The effect of color compensation. This is the same image as Figure 5 after color compensation 
(Eq. (9)) was applied using the color smear matrix shown in Table 1. (A) Color image; (B) red; (C) green; 
(D) blue. Notice that each of the different structures is isolated to the R, G, or B image. 


HS1) color space. The different colored objects in the 
image correspond to separate clusters of points in color 
space, although the clusters corresponding to different 
objects may well overlap. 

The hue and saturation of an object are normally dic¬ 
tated by the light-reactive properties of its dye or stain— 
how it absorbs, reflects, or emits light. The intensity, 
however, is seriously affected by illumination and view¬ 
ing angle. Uneven illumination, for example, normally 
has much more effect on intensity than on the two color 
parameters. Thus it may be productive to segment the 


image in the hue-saturation plane (, Le ., on the color circle) 
rather than in three-dimensional color space, thereby ig¬ 
noring intensity altogether. 

Object measurement and classification 

Once the objects in the image have been identified, 
they can usually be measured in one of the color-channel 
images. Sometimes two or more color-channel images 
can be combined to produce a monochrome image for 
analysis. Further, some objects may be present in different 
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channels and must be analyzed there. In any event, the 
measurement and classification of objects can usually be 
conducted with techniques that are commonly applied 
to monochrome digital images (Castleman, 1996; Pratt, 
1991). 

Discussion 

The digital analysis of color microscopic images is 
becoming more important as new techniques for specimen 
preparation emerge. For example, each of the 24 types of 
human chromosomes can be painted with a distinct color 
by using multicolor fluorescent in situ hybridization (IM¬ 
PISH) techniques (Speicher et ai , 1996, Schrock et al ., 
1996). As the number of colors increases, and as the 
analytical requirements become more stringent, the de¬ 
mands placed on digital image analysis become more 
severe. 

Fortunately the techniques mentioned here for tri-color 
images generalize readily to multispectral digital images 
with two, four, or more primary colors. Then the battery 
of techniques that have been developed for monochrome 
image analysis can be applied to the separate color-com¬ 
ponent images, after they have been balanced, com¬ 
pensated, and perhaps converted to another coordinate 
system. 
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